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(57) Abstract: A method and apparatus for creating edit effects on compressed video data is disclosed. First, an edit point is selected. 
Two anchor pictures on each side of the edit point are then selected. A series of frames is created to create an edit transition at the edit 
1^ poinL The series of frames can be B-pictme frames, I-picture frames, or B-picture friaraes which contain Intra-coded macioblocks. 
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CREATING EDIT EFFECTS ON MPEG-2 COMPRESSED VIDEO 



The invention relates to editing of video content^ and more particularly to a 
method and apparatus for implementing editing transitions on compressed video without 
needing to fully decode and recode the video stream. 



5 

Due to the increase in the demand for video products such as digital cameras, 
camcorders and storage devices (DVDs), digital video editing is becoming increasingly 
popular. Video editing effects are needed to enhance the quaUty of die video production. 
Most video editing can be divided into two major categories: abrapt transitions and gradual 
10 transitions. Gradual transitions include camera movements: panning, tilting, zooming and 
video editing special effects: fade-in. fade-out, dissolving, wiping. Abmpt transition is the 
simplest edit between two shots in which the transition is immediate between two frames. 

Special effects occur gradually over multiple frames. Though, the n\miber of 
possible video special effects is quite high in video production, most of these special effects 
15 fall into several categories, such as fading, dissolving or wiping. During a fade, the intensity 
gradually decreases to, or increases from, a soHd color, hi a dissolve, two shots are additively 
mixed, wherein one increases in intensity, and the other decreases in intensity. Wipes are 
generated by translating a line across the frame in some direction, where the content on each 
side of the line belongs to the two pictures separated by the edit. All these special effects are 
20 used to produce gradual transitions between two scenes. These video editing tools are 
designed for spatial domain processing. 

The large channel bandwidth and memory requirCTients for the transmission 
and storage of image and video necessitate the use of video compression techniques. A 
compression standard referred to as MPEG (Moving Pictures Experts Group) compression is 
25 a set of methods for compression and decompression of ftiU motion video pictures which uses 
an inter-picture compression technique. Intra-pictures are referred to as I-pictures. The inter- 
pictures are divided into two groups: inter-pictures coded using only past reference elements 
which are referred to as P-pictures and inter-pictures coded using a past and/or friture 
reference, referred to as B-pictures. 
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Hence, the visual data in multimedia databases is ejected to be stored mostly in the 
compressed form. Thus, editing of compressed video is also essential. ThCTefore, a typical 
desktop video editing system must first convert the compressed domain representation to a 
spatial domain representation and then perform the editing function on the spatial domain 
5 data. Then, the output of the editing system ndiust be recoinpres^ This decoding, processing 
and subsequent re-encoding is time consuming arid a drain on system resources. 

It is an object of the invention to overcome the above-described deficiencies 
10 by providing a method and apparatus for providing edit effects on compressed video with less 
decoding and re-encoding. The system introduces edit effects without modifying the original 
video streantis by introducing fixed bit patterns between two sequences to generate the effects 
or copy and modify the coded version of the picture wherein all processing is done in the 
compressed domain. 

1 5 According to one embodiment of the invention, a method and apparatus for 

creating edit effects on compressed video data is disclosed. First, an editpoint is selected. 
Two anchor pictures on each side of the edit point are then selected. A series of frames is 
created to create an edit transition at the edit pointv The series of frames can be B-picture 
firames, I-picture firames, or B-picture frames which contain Intra-coded macroblocks. 

20 These and other aspects of the invration will be apparent from and elucidated 

with reference to the embodiments described hereafter. , : 

The invention will now be described, by way of exanxple, with reference to the 
-25 accompanying drawings, wherein: 

Figure 1 illustrates a block diagram of a audio-video ^paratus suitable to host 
embodiments of the invention; 

Figure 2 illustrates a block diagram of a set-top box which can be used to 
implement at least one embodiment of the invention; 
30 Figure 3 is a diagram of a video stream illustrating an edit point according to 

one embodimmt of the invention; 

Figure 4 is a diagram of an edited video stream according to one embodiment 
of the invention; 



BNSCXXyO: <WO__200405760SA1_U> 



wo 2004/057609 . PCT/IB2003/005589 

3 

Figure 5 illustrates a vertical wipe operation according to one embodiment of 

the invention; 

Figure 6 illustrates how maoroblocks along a transition are created from 
original blocks according to one embodim^t of ttte invention; 
5 Figures 7(a)-(b) illustrate circular wipes according to one embodiment of the 

invention; and 

Figures 8(a)-(b) illustrate rectangular wipes according to one embodiment of 

the invention. 

10 

.According to one embodiment of the invention, edit effects on compressed 
video streams are provided with less decoding and re-encoding than conventional methods. 
Such effects can then be included when an edited sequence is played back over a digital 
interface because the output of the edit opaiation is a vahd video stream. The operations can 
15 be generated as part of the int^ace processing and don't need to be created off-line and 
stored on disc. 

The invention will be elucidated by describing an onbodiment of the invention 
where video data is compressed according to the MPEG-2 (Motion Pictures Expert Group) 
standard. According to this standard, a compressed video stream is built up from intra-coded 
20 frames, also known as I-frames, and inter-coded frames. The inter-coded frames can either 
point back to a frame in the compressed video stream, these are so-called B-frames or point 
back as well as forward to frames in the compressed video stream, these are so-called P- 
frames. 

The frames are divided in macroblocks and the inter- and intra-coding as well 
25 as backward and forward pointing is done on macroblock level. MPEG-2 is based on motion 
estimation, meaning a macroblock in a B-frame at a first location in the B-frame can point to 
a second location in a preceding I-frame. 

In one CTibodiment of the invention, it is assumed tibat title first sequence ends 
with a P-fi:ame or an I-frame as the last displayed firame and the second sequence starts with 
30 an I-frame. This can be achieved by ignoring some extra frames. If necessary, it is possible to 
choose the last picture of the first sequence to be an I-firame, by again discarding other 
imwanted pictmres. 

As mentioned above, edit effects can be introduced without modifying the 
original video streams by introducing fixed bit patterns between the two sequences to - 
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generate the effects, or copying and modifying the coded version of the picture wherein all 
processing is perfornied in the (impressed donia^ 

As will be described below, combinations of these two ^preaches are also 
possible in a sin^e traiisition where some macrbblocks are coded using only motion vectors 
5 while others are creiated by copying and modifying the origuoal picturesw Using these 

techniques, standard editing effects such as wipes^ fade out, cross-fade, etc., are provided. 
" Also, other editing effects can be provided that are not normally found in analogue video 
processing but because of the nature of MPEG-2 coding can be generated. 

Figure 1 illustrates an audio-video apparatus suitable to host the invention. 
1 0 The apparatus comprises an input terminal 1 for receiving a digital video signal to be 

recorded on a disc 3. Further, the apparatus comprises an output terminal 2 for supplying a 
^' digital video signal reproduced firom the disc. These terminals may in use be coimected via a 
digital interface to a digital television receiver a^d decoder in the form of a set-top box (STB) 
12, which also receives broadcast signals from satellite, cable or the like, in MPEG TS 
15 format. The set-top box 12 provides display signals to a display device 14, which may be a 
conventional television set. 

The data area of the disc 3 consists of a contiguous range of physical sectors, 
having corresponding sector addresses; This address space is divided into sequence areas, 
with a sequence area being a contiguous sequerice of sectors. The video recording apparatus 
20 as shown in Figure 1 is composed of two major system parts, namely the disc subsystem ^ 
and the video recorder subsystem 8, controlling/both recording and playback. The two 
' subsystems have' a liumber of features, as will be readily understood, including that the disc 
subsystem can be addressed traiisparently m temis of to 

guarantee a maximum sustainable bit-rate for reading and/or writing data from/to the disc. 

25 Suitable hardware arrangements for implem^ting such an apparatus iare 

known to one skilled in the art, with one example illustrated in patent ^>plication WO-A- 
00/00981 . The ^paratus g^erally comprises signal processing units^ a readAvrite unit 
including a readAvrite head configured for reading from/writing to a disc 3. Actuators 
position the head in a radial direction across the disc, while a motor rotates the disc. A 

30 microprocessor is present for controlling all the circuits in a known manner. 

Figure 2 shows an embodiment of the apparatus in accordance with the 
invention. The apparatus comprises an input terminal 1 for receiving an information signal 
and a signal processing imit 100. The signal processing unit 100 receives the video 
information signal via the input terminal 1 and processes the video information into an . 
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information file for recording the information jBle on the disc 3. Further, a read/write unit 102 
is available. The read/write unit 102 comprises a read/write head 104, which is in the present 
example an optical read/write head for reading/writing the information file on/firom the disc 
3. Further, positioning means 106 are present for positioning the head 104 in a radial 
5 direction across the disc 3. A read/write amplifier U08 is present in order to amplify the signal 
to be recorded and amplifying the signal read from the disc 3. A motor 1 10 is available for 
rotating the disc 3 in response to a motor control signal supplied by a motor control signal 
generator imit 1 12. A microprocessor 1 14 is present for controlling all the circuits via control 
lines 1 1 6, 1 1 8 and 120. An input unit 1 30 allows a user to select an edit point in the video 
10 data where the edit transition will be added. 

The signal processing unit 100 is adapted to convert the video data received 
via the input terminal 1 into blocks of information in the channel signal: the size of the blocks 
of information can be variable but may, for example, be between 2 MB and 4 MB. ITie write 
unit 102 is adapted to write a block of information of the channel signal in a sequence area on 
IS die disc 3. The information blocks corresponding to the original video signal are written into 
many sequence areas that are not necessarily contiguous, which is known as fragmented 
recording. As will be desoibed below, the signal processing unit 100 creates the edit 
transitions in accordance with the various edit operations. 

According to one embodiment of the invention, a transition between two 
20 pictures is created by inserting new pictures which use motion vectors which reference the 
original pictures. The inserted new pictures are B-pictures and therefore can refer to the old 
anchor picture, tibiie new anchor picture or both pictures. Because motion vectors are defined 
per macroblock, each macroblock can be chosen from either the old picture, the new picture 
or a combination of both. 
25 If the original sequence includes B-pictures then there is a problem if a 

sequence of B-pictures is inserted. Because of the picture re-ordering, the final I/P picture in 
the first sequence will be displayed after ftto ins^ed B-pictures. To deal with this, the first I- 
picture of the second sequence can be placed before the set of inserted pictures. For example, 
siqppose the original stream is as illustrated in Figure 3, wherein the edit point occurs between 
30 Bi5 and to- According to the invration, the I-picture I20 is inserted before the edit point and 
the edit transition is generated by frames Bxi Bx2 - • • Bxn which is illustrated in Figure 4. 

By generating a sequence of B-pictures with specific motion vectors, a 
transition can be created between the two original pictures. The sequence of B-pictures to be 
inserted is independent of the content of the original pictures and so the same sequencei>f B- 
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pictiires will generate the same effect independent of the content of the original pictures. The 
size of the iiiserted B^pictures?will be very small resulting in a low average bit rate. 

A wipe operation is a transition from one picture to another which can be 
p^ormed' horizontally, vertically or diagonally . For example. Figure 5 illustrates a vertical 
5 wipe. To perform a wipe in MPEG-2, each macroblock is chosen from either the first or the 
second anchbr picture, Because of restrictions in MPEG-2 coding (mot^^ 
^ point outside the coded part of the picture), the wipe is implemented block by block. This 
kind of wipe effect can be implemented using B-pictures to choose the blocks from either the 
first or second anchor picture. 
10 F6r example^ toimplonent aw 

illustrated in Figure 5, initially all the blocks are taken from the first anchor picture 502. Then 
^ af the next stage in the wipe; all left-niost macroblocks are taken from the second anchor 
picture 504 and the rest of the macroblocks are taken from the first anchor picture 502. In the 
next step an additional row of blocks are chosen from the second anchor picture 504 and so 
15 on until the first anchor picture 502 has been rq)laced by the second anchor picture 504. By 
repeating the inserted B-pictUres a nmnber of times, the speed of the wipe can be controlled. 

There are several variations of this wipe effect. In the first variation, the 
second anchor picture replaces the first anchor picture but all blocks are shown in their 
normal position on the screen (and so all motion vectors are zero). The second variation is 
20 performed by showiiig the rightmost colimm of blocks from the second anchor picture in the 
left most column and then the blocks from the second anchor picture.push across tiie first 
anchor picture. Similaily, tibe second anchor picture can appear to push the first anchor 
picture off the screen, Le., the blocks move one position to the. right for each iteratioii. 
Variations of fliis are also possible, e.g., the hew picture sqipears to push across while the old 
25 picture £q>pears stationary. — 

Li this illustrative 'exanq>le, the wipe is performed on a block by block basis 
and not in a smootii pixel-by-pixel basis. Another variation is to use bi-directional B-pictures 
to merge the blocks of the old and new pictures, i.e., B-picture then points to a block from the 
first picture and a block finom the second picture, this means that during the wipe the blocks 
30 are merged before the second picture rq^laces the first. Similarly, these wipe efifects can be 
done in the horizontal direction. 

OthCT wipe variants are also possible, e.g., wipe from both left and right (or 
top and bottom) and meet in the middle. Li addition, a wipe can start in a top com^ and 
expand through the comiplete picture in a diagonal manner. It is also possible to wipe tiie 
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even macroblock tows fixmi the left and then the odd macroblock rows from the right or do 
both in parallel from opposite directions. Similar operations can also be performed for 
horizontal wipes. 

For cellular automata style transitions, a few blocks from the old picture are 
5 replaced with blocks from the new picture. Then, based on a predetermined rule fiuther 

blocks are replaced on each successive iteration. For example, a replacement mle could be a 
rule where any block that is adjacent to an already replaced block is replaced. This gives the 
impression of the new picture growing out of the old picture. This can be perfoitaied using 
motion vectors of size zero pointing to either the old picture or the new picture. In this way, 
10 the same block in the other picture is chosen. A variation of this operation is where the block 
in the old picture is replaced by a combination of the block at the same location in both the 
old and new picture (done by having two motion vectors which are both zero) and then in the 
next iteration it is rq[>laced by the block from the second picture. 

For efiFects using manipulation of intra-coded pictures, transitions are 
1 5 generated by copying and manipulating the intra-coded blocks. This can involve 

manipulating the old and new pictures independently or combining the two together. The two 
original pictures should both be I-pictures and the inserted pictures are also coded as I- 
pictures. The discrete cosine transform (DCT) coefficient blocks of the two original pictures 
are manipulated to cause edit transitions. While this embodiment involves VLC decoding and 
20 encoding, the coding has a much lower complexity than fall MPEG-2 decoding and 

encoding. Inserting a sequence of I-frames may increase the bit-rate but some solutions are: 
insert empty P-frames to cause copying and reduce the average bit-rate and also slow down 
the speed of the fade; increase tiie quantiser scale of the I-frame to reduce tiie coded bits. In 
general, the edit effect involves transitions to pictures that can be easily coded so the numba: 
25 of bits required wiU be less later in the transition. 

A fade out operation can be performed by copying the I-frame a number of 
times and each time reducing tiie size of all coefficients by a predetermined frictor, wherein 
the size of the reduction determines the speed of the transition. As the picture fades out, the 
number of bits needed should reduce very quickly. Similarly, a fade-in operation is 
30 performed in the opposite way. Fade out can also be combined with other effects. Fade out 
with blurring can be achieved by throwing away the higher frequency components in the 
macroblocks. Fade to Black-and-White followed by fade out can be achieved by iBrst 
(gradually) reducing the chroma components before starting to reduce the luminance 
component. 
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: For a cross-fade jopCTation, a smooth transition from the first sequence to flie 
. second sequence is generated. As with fade to Black, cross fade can be performed by 

opiating on the DCT coefificieats of the I-frames. Basically, the DCT coefficients from the 
. two I-frames are added as follows: a*DCTi+(l-a)*DCT2 where a starts at 0 and progresses to 
5 1 . The duration of the transition can be changed by choosing the speed to increase the 
■ ■ coefficiient a. -f. 

For a DC Cross-Fade operation, the old picture is faded to a DC only value, 
. i:e., in each successive picture more AC coefficients are removed. In addition, a factor of the 
i DC coefificimt of the new picture can be added so the result is the average of the two DC 
10 .values. Then, flie DC coefficient of the first picture can be faded out while adding the AC 
: coefficients of the new picture. Variations of this operation can be created by performing this 
with the chromenence (U,V) coefficients first or else fading these to a specific value. A third 
variation is to fade first between ttie U,V coefficients using (a, so that the old picture 
luminance is combined with the new picture chromenence and then fade to the new picture 
15 luminancie. 

li is also possible to create edit effects that combine using both motion vectors 
and the manipulation of intra coded blocks at the same time. In this case, the inserted pictures 
will be B-pictures with some Intra-coded macroblocks. As described above, a wipe occurs 
where a transition from one picture to another is performed either horizontally or vertically. 
20 Figure 6 illustrates how macroblocks along the transition are created from the original blocks. 
The chosen parts of the pictures from the first sequence 601 and the second sequence 602 are 
combined (in the decoded domain) and the blocks are re-^coded as Intra-coded blocks. 
; : OthCT macroblocks are copied directly firom the previous or next picture. Because the MPEG- 
2 standard explicitly excludes motion vectors that point outside the picture, it is necessary to 
25 re-encode (intra) these blocks along the transition 604. As illustrated in Figure 6» there is a 
hard break 603 between the two pictures. It is also possible to have some overlapping pixels 
between the two^ pictures to give an averaging effect along the transition. 

Several variations of this wipe are possible. In the first case, the new picture 
pushes the old picture fix>m the screen. In the second case, the new picture ovCTwrites flie old 
30 picture but there is no change in the position on screen of the old picture. Other wipe variants 
are also possible, e.g., wipe from both left and right (or top and bottom) and meet in the 
■ middle. Wipe from top comer and expand through complete picture. It is also possible to 
wipe the even macroblock rows from Uxe left and the odd macroblock rows from the right or 
do both in parallel fix>m opposite directions. 
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For circular wipes, the new picture appears from a point in the center and 
replaces the old picture by outwardly expanding circles as illustrated in Figure 7(a). The 
opposite case where the new picture appears in a circle on the edges and moves to a point is 
also possible and is illustrated in Figure 7(b), It will be understood that when these images 

5 are displayed, the transition may in some cases appear elliptical and not circular. 

For macroblocks either completely inside or outside the circle, there is no 
problem, they will be taken from either the old picture or new picture by using vector motion. 
For macroblocks on the circle, it is necessary to decode the two blocks and then choose the 
appropriate pixels from the old and new pictures to create the circular effect and then re- 

10 encode the block as an Intra-coded block. By re-encoding the blocks on the circle, a clean 
break at the transition point can be achieved. It is also possible, for example, to just combine 
the two blocks (using motion vectors to both blocks) to give an un-smooth blurred transition. 
A similar effect but with the new picture coming from a point in the middle but expanding as 
a rectangle and starting at the edges and moving inward to the center is also possible as 

15 illustrated in Figures 8(a)-(b). Again in this case, the blocks around the border must be re- 
encoded to get a clean break. 

It will be understood that the different embodiments of the invention are not 
limited to the exact order of the above-described st^s as the timing of some steps can be 
interchanged without affecting the overall operation of the invention. Furthermore, the term 

20 "comprising^ does not exclude other elements or steps, the terms "a'* and "an*' do not exclude 
a pluraUty and a single processor or other unit may fulfill the functions of several of the units 
or circuits recited in the claims. 

The invention can be sununarised as method and ^paratus for creating edit 
effects on compressed video data is disclosed. First, an edit point is selected. Two anchor 

25 pictures on each side of the edit point are then selected. A series of frames is created to create 
an edit transition at the edit point The series of frames can be B-picture frames, I-picture 
frames, or B-picture frames which contain Intra-coded macroblocks. 
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1 . A method for creating edit effects on compressed video data, comprising the 
steps of: 

selecting an edit point; 

selectihg two anchor pictures on each side of the edit point; 
5 creating a series of firames to create an edit transition at the edit point. 

2. The method according to claim 1 , wherein the series of frames are B-pictm-e 
frames. 

10 3. The method according to claim 2, wherein the series of B-picture frames 

reference a JSist anchor picture, a second anchor picture or a combination of the first and 
second anchor pictures. 



15 



4. The method according to claim 2, wherein the edit effect is a wipe. 

5. The method according to claim 2, wherein the edit effect is a fade. 



6. The method according to claim 3, wherein each macroblock of the edited 
image is chosen jfirom the first anchor picture, the second anchor picture or a combination of 

20 the first and second anchor pictures. 

7. The method according to claim 6, whorein motion vectors are defined on a per 
macroblock basis. 

25 8. The method according to claim 4, wherein the wijpe efifect is created by 

selecting macroblocks on a first side of the transition from the first anchor picture and 
selecting macroblocks on a second side of die transition fix)m the second anchor picture. 
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9. The method according to claim 8, wherein macroblocks on the second side of 

the transition are shown in a location which corresponds to their final position in a resulting 
picture. 

5 10. The method according to claim 8, wherein the maeroblocks of the second 

anchor picture appear to push across the first anchor picture. 

1 1 . The method according to claim 8, wherein the second anchor picture appears 
to push the first anchor picture off a screen. 

10 

12. The method according to claim 4, wherein the wipe is performed in a vertical, 
horizontal or diagonal direction. 

1 3. The metitiod according to claim 12, wherein the wipe is started on two sides of 
1 5 the first anchor picture and meet in the middle of the first anchor picture. 

14. The method according to claim 6, wherein macroblocks firom the first anchor 
picture are randomly replaced by corresponding blocks in the second anchor picture. 

20 15. The method according to claim 6, fijrther comprising the steps of: 

randomly replacing macroblocks fi-om the first anchor picture with a 

combination of the corresponding macroblocks in the first and second anchor pictures; 

replacing the combination macroblocks with the corresponding macroblocks 

firom the second anchor picture. 

25 

16. The me&od according to claim 1, wherein the s^es of firames are I-picture 
frames. 

17. The method according to claim 16, wherein DCT coefficient blocks of the two 
30 anchor pictures are manipulated in the series of I-picture fi*ames to create the edit transition. 

18. The method according to claim 17, wherein the edit efifect is a fede. 
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1 9. The method according to claim 1 8, wherein the fade effect is created by 

reducing the size of all DCT coefficirats by a predetermined factor in each successive I- 
picture frame of the edit transition. 

5 20; The method according to claim 1 7, wherein DCT coefficients of a first anchor 

picture are reduced to zcto iand DCT coefficients of a second anchor picture are increased 
frorri zero to their actual values in each successive I-picture frame of the edit transition. 

21 . The method according to claim 1 , wherein the series of frames are B-picture 
1 0 fi^es which may contain Intra-coded macroblocks. 

22. The method according to claim 2 1 , whwein the Litra-coded macroblocks are 
for macroblocks which form a transition between the images of the two anchor pictures. 

15 23. An apparatus for creating edit effects on compressed video data, comprising: 

means (130) for selecting an edit point; 

means (100) for selecting two anchor pictures on each side of the edit point; 
means (100) for creating a series of frames to create an edit transition at the 

edit point. 

20 
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